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Abstract 



Background/Context. Despite broad consensus in science education on the goal of teaching for 
deep conceptual understanding, there is widespread disagreement about the role of teachers in 
curriculum. There have been calls for high-quality curricula developed by experts that work 
regardless of teachers’ level of expertise (e.g., Bruner, 1960). Others have called for professional 
development that builds teachers’ expertise in designing and defining curriculum (e.g., Singer, 
Krajcik, Marx, & Clay-Chambers, 2000). To date, there has been little compelling evidence from 
experimental studies as to which approach leads to better instruction and higher student 
achievement. 

Purpose. The purpose of this study is to compare the efficacy of three different approaches to 
professional development in Earth science education: preparing teachers to adopt expert-design 
curricula; preparing teachers to use a principled approach to design curricula; and preparing 
teachers to use a principled approach to adapt curricula. The claims examined through the study 
are that one or more of these approaches is more effective than the other(s) in improving 
teaching and learning. 

Population. A total of 56 6 th , 7 th , and 8 th grade teachers from 19 middle schools in a large urban 
district participated. Of these, 14 teachers were assigned to the IES condition, 13 to the ESBD 
condition, 14 to the IES condition, and 15 to the control group. There were no significant 
differences across conditions with respect to years of teaching experience, or content preparation 
in Earth science. Teachers in the study were all volunteers, as is common in professional 
development; therefore, the study findings may not generalize to teachers who are forced by their 
district or school leaders to participate. 

Professional Development Interventions. There were three different approaches to professional 
development tested in the study, which were compared to a fourth, business-as-usual control 
condition. All three intervention conditions were based on the principles of Understanding by 
Design (Wiggins & McTighe, 1998), an approach to curriculum development that focuses on 
teaching for and assessing conceptual understanding: 

Earth Science by Design (ESBD) is a professional development intervention that provided 
teachers with training in how to develop high-quality curriculum units aligned to local 
standards and gave them time to develop these units, along with providing follow-up 
mentoring. 

The Investigating Earth Systems (IES) curriculum included structured professional 
development that prepared teachers to use curriculum units based on the principles of 
Understanding by Design. In contrast to the ESBD condition, experts in Earth science and 
curriculum development designed the instructional materials. 

A Hybrid condition provided teachers with training in how to develop high-quality 
curriculum units aligned to local standards and gave them high-quality materials to revise 
and adapt in developing units. 

In the Control condition, teachers received no special professional development, but were 
expected to teach to the same standards as teachers in the other three conditions. 

Professional development questionnaires were administered after the workshops and at the end 
of the school year. Data from these questionnaires suggests that the professional development 




providers implemented their models in ways consistent with their philosophies and with their 
initial plans. In addition, the data suggest that teachers in all three of the treatment conditions 
were able to follow instructions regarding unit design, adoption, or adaptation as specified by the 
professional development providers in the workshops. 

Data Collection and Analysis. Primary sources of data for the study were standards-based tests 
developed by the researchers, analyses of naturally occurring teacher assignments, and classroom 
observations. We focus in this presentation on results from the tests and teacher assignments. 



Standardized Tests. Each student took a pretest and posttest focused on the content their Earth 
science unit was supposed to cover, according to the district’s standards. We developed unit tests 
of 20 and 22 items aligned to the district’s standards for Earth science. We performed both 
classical and IRT -based item analysis of student responses. Test reliabilities (Cronbach’s a) 
ranged from 0.69-0.78. To examine impact across grades, we rescaled all test scores from 0-100. 
We included grade level and terms for grade X condition interactions in the models for overall 
treatment impact to control for possible differences associated with grade. Those models yielded 
the same results as the simpler model below: 
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Analyses of Teacher Assignment Data. We collected three naturally occurring teacher 
assignments and a performance assessment from each teacher. Validation studies showed 
positive associations between the quality of naturally-occurring assignments and the work 
students produced (Matsumara, Patthey-Chavez, Valdez, & Gamier, 2002; Shkolnik et al., 2007) 
and standardized test scores (Matsumara, Gamier, Pascal, & Valdes, 2002; Newmann, Bryk, & 
Nagaoka, 2001). Ours is one of the first studies to examine assignment quality in science, rather 
than language arts and mathematics assignments. 

The study team developed five 4-point rubrics to analyze teacher assignments and the 
performance assessment. The rubrics addressed these dimensions: scientific communication, 
construction of knowledge; quality of Earth science content; approach to the nature of science; 
and opportunity for student inquiry. Coders double-scored all assignments; overall inter-coder 
agreement within one point of the rubric scales was 87 percent. The study team used the Many- 
Facet Rasch Model to combine scores across rubrics, controlling for rater severity and rubric 
difficulty. 

We used HLM 6.0 software to model the impact of treatment condition on overall assignment 
quality. Because our study is a multi-site, cluster randomized trial with assignment at the teacher 
level, we modeled assignments at Level 1, treatment effects at Level 2, and school effects at 
Level 3. The treatment effect was modeled as a random effect that was allowed to vary between 
schools. For results depicted in the graph below, the model tested was: 
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Findings. Student Learning. With grade levels combined, differences in percentage point gains 
from pretest to posttest were statistically significant for ESBD and Hybrid conditions compared 
with IES and Control conditions. ESBD students achieved the highest gain, 15%, followed by 
Hybrid students, 14%. IES gains were 10% followed by Control at 9%. Figure 1 summarizes 
these findings. 

Insert Figure 1 here. 



Teacher Assignment Quality. Assignments of teachers who were randomly assigned to the IES 
and Hybrid conditions were judged to be of significantly higher quality than assignments 
submitted by teachers assigned to the control condition. 



Insert Figure 2 here 

Conclusions and Implications. The study results suggest that professional development aimed 
at helping teachers to adapt high-quality curriculum materials using the Understanding by Design 
approach is the most effective approach for improving Earth science teaching and learning in the 
middle grades. Although all three professional development interventions were effective in 
improving teaching or learning on some dimensions, the Hybrid classrooms in the study had the 
most consistent results across measures of impact on teaching and learning. 

The fact that students in the IES condition did not perform as well on our student learning 
measures as their assignment quality might indicate does not detract from these findings. The 
IES teachers could not supplement their instruction with materials that addressed standards and 
topics not in the IES materials. This fact does not diminish the significance of the results; 
instead, it points to the critical need for curriculum developers to design materials for teachers to 
adapt in a principled way. 

From a policy perspective, the study suggests that neither simply providing teachers with high- 
quality curricula nor providing them with resources for designing curricular units on their own is 
as effective as designing curriculum for adaptation. Our results suggest specific professional 
development activities that can help teachers adapt curriculum. The Understanding by Design 
approach prepares teachers to select target understandings that are at the core of what students 
need to know and develop or select activities designed to develop those understandings. The 
study provided evidence that teachers used these elements of their professional development to 
affect their teaching practice and, consequently, student learning. 
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Figure 1. Impact of the professional development interventions on student achievement 
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Figure 2. Impact of the professional development interventions on instruction 



